Skip to content

feat: add storage tier change events script#118

Merged
lhoupert merged 2 commits into
mainfrom
feat--add-storage-tier-change-events-script
Jul 8, 2026
Merged

feat: add storage tier change events script#118
lhoupert merged 2 commits into
mainfrom
feat--add-storage-tier-change-events-script

Conversation

@lhoupert

@lhoupert lhoupert commented Mar 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

branched from #103 (now rebased on main; the #103 CLI-command commit is merged, so this PR is just the submitter script + the --date-field extension below)

  • Adds submit_storage_tier_workflows.py operator tool that queries STAC in 24h windows over a date range and submits one batch-change-storage-tier webhook payload per window to trigger the eopf-storage-tier-batch-job WorkflowTemplate deployed in https://github.com/EOPF-Explorer/platform-deploy/pull/130 via Argo Events
  • Adds unit tests (covering time-window generation, STAC querying, dry-run, error handling, and payload structure)
  • Updates operator-tools/README.md with full usage docs and a tool-selection reference table

Update: --date-field (window on registration time)

The submitter originally windowed only on sensing datetime. To drain the S2 backlog by registration time (bulk-converted items were registered long after they were sensed), this adds:

  • --date-field {datetime,created,updated} (default datetime, backward compatible). Non-default fields swap the pystac-client datetime= range for a CQL2 between filter on the chosen property — same pattern scripts/query_stac.py uses to harvest by updated.
  • --storage-class default flipped STANDARD_IASTANDARD (STANDARD_IA is unused in the deployment; documented commands stay explicit).
  • timeout=30 on the webhook POST (ruff S113, on this script's own code) so a hung endpoint can't stall the per-window loop.
  • README gains the registered-window backlog example.

Planned use (#182 follow-up): a one-off prod run over sentinel-2-l2a, windowing on --date-field created for items registered 2025-11-01 → (today − 3 months), moving their S3 objects to STANDARD. Dry-run first, then live via port-forward. The Sensor + batch template are date-agnostic (they consume resolved item_ids), so no cluster-side change is needed for this capability.

Why

Changing S3 storage classes for thousands of items over a multi-month date range is too slow when done sequentially on a local machine. The script fans work out to Argo: one workflow per 24h window, with intra-window parallelism handled by the eopf-storage-tier-batch-job template. Empty windows are skipped automatically. Windowing on created lets us target the registration backlog that a sensing-date window would miss.

Test plan

  • uv run --group test pytest tests/unit/test_submit_storage_tier_workflows.py passes (incl. new --date-field / CQL2 / default-storage-class cases)
  • uv run --group test pytest tests/unit/test_change_storage_tier_commands.py passes
  • Full pre-commit gate green (ruff / ruff-format / mypy / pytest)
  • --date-field created builds a CQL2 between on created; default datetime path unchanged (asserted via mocked Client.search kwargs)
  • Dry run against staging collection
  • Live run (port-forward active) submits correct payload shape and logs action: batch-change-storage-tier with item IDs per window

@lhoupert

Copy link
Copy Markdown
Collaborator Author

I tested yday with the command:

 python operator-tools/submit_storage_tier_workflows.py \
    --start-date 2026-03-08 \
    --end-date 2026-03-12 \
    --collection sentinel-2-l2a-staging-backup-20260312 \
    --storage-class STANDARD_IA \
    --webhook-url http://localhost:12001/samples

and it worked well:

image

lhoupert and others added 2 commits July 8, 2026 13:02
The batch storage-tier submitter only windowed on sensing `datetime`, so it
could not target the S2 backlog by *registration* time (`properties.created`).
Bulk-converted items were registered long after they were sensed, so a
sensing-date window misses them.

Add `--date-field {datetime,created,updated}` (default `datetime`, backward
compatible). Non-default fields swap the `datetime=` range for a CQL2 `between`
filter on the chosen property — the same pattern `scripts/query_stac.py` uses
to harvest by `updated`.

Also:
- flip `--storage-class` default `STANDARD_IA` -> `STANDARD` (STANDARD_IA is
  unused in the deployment; documented commands stay explicit regardless);
- add `timeout=30` to the webhook POST so a hung endpoint can't stall the
  per-window submission loop (ruff S113, on this script's own code);
- document the registered-window backlog example in operator-tools/README.md.

Plan: claude-docs/plans/s2_tier_standard_backlog.md (Task 2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lhoupert lhoupert force-pushed the feat--add-storage-tier-change-events-script branch from 5997970 to bc65a8a Compare July 8, 2026 12:16
@lhoupert lhoupert temporarily deployed to harbor-registry July 8, 2026 12:16 — with GitHub Actions Inactive
@lhoupert lhoupert temporarily deployed to harbor-registry July 8, 2026 12:16 — with GitHub Actions Inactive
@lhoupert lhoupert merged commit 1639baf into main Jul 8, 2026
10 checks passed
@github-actions github-actions Bot mentioned this pull request Jul 3, 2026
lhoupert added a commit that referenced this pull request Jul 13, 2026
Brings in the two commits exclusive to main:
  #118 (1639baf) storage-tier change-events script
  #278 (017898d) eodash layerExclusive + style link for S2 L2A

Conflicts resolved toward the integration branch (phase5 = source of truth):
  - operator-tools/README.md tool table: kept phase5's 'uv run python'
    modernization for the shared rows; added #118's three submit_* rows
    (normalized to 'uv run python').
  - tests/test_stac_collections.py (add/add): unioned — kept phase5's S1 GRD
    RTC collection asset tests AND main's S2 L2A eodash metadata tests
    (no function-name overlap).

Net change vs phase5 == exactly #118 + #278; no phase5 work reverted.
806 passed, ruff clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants